Wiki Search

In this example, we'll show you different ways of performing a vector search. We'll start by constructing a basic search toolbar using Python and the Vecto SDK. Then, we'll demonstrate how to perform this search via the command line using our documentation website.

A wiki search toolbar could be a useful addition to your personal website or an application you're developing, allowing users to search for Wikipedia articles directly.

Here's what we'll be covering in this tutorial:

1. Creating a Vector Search Toolbar in Python: We'll start by using the Vecto Python SDK to perform a vector search. We'll guide you through initializing the Vecto class and performing a search using ipywidget. Then, we'll also provide an example of the syntax you'd use to create your own lookup function.

2. Performing a Vector Search using cURL: For those not using Python, we've added this short section on how to perform cURL requests using our Swagger UI in the documentation.

By the end of this tutorial, you will have a solid understanding of how to build a basic Wiki search toolbar using Python. So, let's get started!

Getting the Wiki Key

We've setup a vector space which comes preloaded with Wikipedia data for your convenience. To request access, simply visit this link:

https://www.vecto.ai/request-free-wikipedia-db-access

You should get the vector space id, as well as your own token.

Set Up Vecto Application in Python

!pip install ftfy tqdm ipywidgets==7.7.1 vecto-sdk

Initialize Vecto

We'll begin by initializing the Vecto class and supplying it with our wiki_search Vector Space ID and authentication token.

By default, the Vecto class checks for the existence of VECTO_API_KEY in the environment. If it doesn't exist, you can directly supply the token parameter. Since we're going to interact with the newly created vector space, it's also essential to provide its ID.

Replace the placeholders with the actual values for the token and vecto_space_id, then run the cell.

from vecto import Vecto
import os

# token =  os.environ['VECTO_API_KEY']
token = ""
vector_space_id = ""

vs = Vecto(token, vector_space_id)

Smart Search using Text Vector Search

Let's run a lookup function and hook it up with ipywidgets.

from ipywidgets import interact_manual, IntSlider, FileUpload
from IPython.display import display
import io

def text_query(query, top_k=10):
    f = io.StringIO(query)
    response = vs.lookup_text_from_str(query, top_k)
    display_results(response)
    
def display_results(results):
    output = []
    for result in results:
        formatted_result = f"Title: {result.attributes['title']}\nText: {result.attributes['text']}\nURL: {result.attributes['url']}\nSimilarity: {result.similarity}\n\n"
        output.append(formatted_result)
    print(*output, sep='\n')

You can then perform searches using text queries. Using the interactive cell widget:

• Type your query text in the available text box.
• Select the number of results with the highest search similarity to view top_k.

interact_manual(text_query, query="how to cook chicken", top_k=IntSlider(min=1, max=50))

When using Vecto, the results you get should correspond to the search query you've inputted.

Show Expected Output

{
  "results": [
    {
      "attributes": {
        "title": "Chicken as food",
        "text": "Chicken can be cooked in many ways. It can be made into sausages, skewered, put in salads, traditionally grilled or by using electric grill, breaded and deep-fried, or used in various curries. There is significant variation in cooking methods amongst cultures. Historically common methods include roasting, baking, broasting, and frying. Western cuisine frequently has chicken prepared by deep frying for fast foods such as fried chicken, chicken nuggets, chicken lollipops or Buffalo wings. They are also often grilled for salads or tacos.",
        "url": "https://en.wikipedia.org/wiki?curid=5741239"
      },
      "id": 1358105,
      "similarity": 0.6214650869369507
    },
    {
      "attributes": {
        "title": "Fried chicken",
        "text": "Pan frying (or shallow frying) requires a frying pan of sturdy construction and a source of fat that does not fully immerse the chicken. The chicken pieces are prepared as above, then fried. Generally, the fat is heated to a temperature hot enough to seal (without browning, at this point) the outside of the chicken pieces. Once the pieces have been added to the hot fat and sealed, the temperature is reduced. There is debate as to how often to turn the chicken pieces, with one camp arguing for often turning and even browning, and the other camp pushing for letting the pieces render skin side down and only turning when necessary. Once the chicken pieces are close to being done, the temperature is raised and the pieces are browned to the desired color (some cooks add small amounts of butter at this point to enhance browning). The moisture from the chicken that sticks and browns on the bottom of the pan becomes the fonds required to make gravy.",
        "url": "https://en.wikipedia.org/wiki?curid=18598020"
      },
      "id": 1041067,
      "similarity": 0.5921482443809509
    },
    {
      "attributes": {
        "title": "Roast chicken",
        "text": "Oven roasted chicken is often served in the United States for special family meals including holidays such as Rosh Hashanah, Christmas and sometimes Easter or Thanksgiving. Considered a \"comfort food\" by many, oven roasted chicken had a resurgence of popularity in the mid to late 1990s as more restaurants and recipe publishers started to refocus on classic American fare. The basic roasting process involves removing the neck and giblets from the cavity, trussing the bird and folding the wings underneath, seasoning the skin and/or cavity, and then placing the bird in a pre-heated oven. The bird should be basted regularly, and is considered done when a meat thermometer registers 170°F for white meat or 185°F for dark meat. It is commonly stuffed with oyster stuffing, chestnut stuffing or potato stuffing, or other varieties of stuffings.",
        "url": "https://en.wikipedia.org/wiki?curid=800845"
      },
      "id": 6619927,
      "similarity": 0.5752872228622437
    },
    {
      "attributes": {
        "title": "Roast chicken",
        "text": "Roast chicken is chicken prepared as food by roasting whether in a home kitchen, over a fire, or with a rotisserie (rotary spit). Generally, the chicken is roasted with its own fat and juices by circulating the meat during roasting, and therefore, are usually cooked exposed to fire or heat with some type of rotary grill so that the circulation of these fats and juices is as efficient as possible. Roast chicken is a dish that appears in a wide variety of cuisines worldwide.",
        "url": "https://en.wikipedia.org/wiki?curid=800845"
      },
      "id": 6619919,
      "similarity": 0.5664731860160828
    },
    {
      "attributes": {
        "title": "Fried chicken",
        "text": "A pressure cooker can be used to accelerate the process. The moisture inside the chicken becomes steam and increases the pressure in the cooker, such that lowering the cooking temperature is needed. The steam also cooks the chicken through, but still allows the pieces to be moist and tender while maintaining a crisp coating. Fat is heated in a pressure cooker. Chicken pieces are prepared as described above and then placed in the hot fat. The lid is placed on the pressure cooker, and the chicken pieces are thus fried under pressure. The original recipe used by the KFC franchise used this method, which was marketed as \"broasting\" by the Broaster Company.",
        "url": "https://en.wikipedia.org/wiki?curid=18598020"
      },
      "id": 1041069,
      "similarity": 0.561736524105072
    },
    {
      "attributes": {
        "title": "Barbecue chicken",
        "text": "In North America, barbecue chicken is often seasoned with a spice rub, then coated with a tomato based barbecue sauce, and grilled. Some versions only use a spice rub and don't use sauce. Barbecue chicken can also be prepared in pressure cookers, in which the chicken is cooked inside the cooker with barbecue sauce, and in slow cookers.",
        "url": "https://en.wikipedia.org/wiki?curid=23778744"
      },
      "id": 7230003,
      "similarity": 0.5588650703430176
    },
    {
      "attributes": {
        "title": "Tandoori chicken",
        "text": "The marinated chicken is placed on skewers and cooked at high temperatures in a tandoor oven, which is heated with charcoal or wood, which adds to the smoky flavour. The dish can also be cooked in a standard oven, using a spit or rotisserie, or over hot charcoal.",
        "url": "https://en.wikipedia.org/wiki?curid=744008"
      },
      "id": 2581757,
      "similarity": 0.5547412633895874
    },
    {
      "attributes": {
        "title": "Chicken as food",
        "text": "Chicken can be prepared in a vast range of ways, including baking, grilling, barbecuing, frying, and boiling. Since the latter half of the 20th century, prepared chicken has become a staple of fast food. Chicken is sometimes cited as being more healthful than red meat, with lower concentrations of cholesterol and saturated fat.",
        "url": "https://en.wikipedia.org/wiki?curid=5741239"
      },
      "id": 1358084,
      "similarity": 0.5387617945671082
    },
    {
      "attributes": {
        "title": "Fried chicken",
        "text": "Deep frying requires a deep fryer or other devices in which the chicken pieces can be completely submerged in hot fat. The process of deep frying is placing food fully in oil and then cooking it at a very high temperature. The pieces are prepared as described above. The fat is heated in the deep fryer to the desired temperature. The pieces are added to the fat and a constant temperature is maintained throughout the cooking process.",
        "url": "https://en.wikipedia.org/wiki?curid=18598020"
      },
      "id": 1041068,
      "similarity": 0.5387256741523743
    },
    {
      "attributes": {
        "title": "Rotisserie chicken",
        "text": "Rotisserie chicken is a chicken dish that is cooked on a rotisserie by using direct heat in which the chicken is placed next to the heat source.",
        "url": "https://en.wikipedia.org/wiki?curid=46247445"
      },
      "id": 4370253,
      "similarity": 0.522800624370575
    }
  ]
}

Compare and contrast the result if you've entered "how to cook chicken" in the default wikipedia search. You will likely receive an error saying that the page doesn't exist. This is because Wikipedia employs a keyword-based search system.

However, Vecto employs a more sophisticated approach. It isn't constrained by specific keywords and can provide relevant results even when your query doesn't match an exact Wikipedia page title. With Vecto, you're able to explore more diverse and contextually relevant results that better match your information needs.

Try and experiment with other fun search queries, for instance, proof of aliens. Vecto can still provide these recommendations, which highlights Vecto's capability to generate intelligent suggestions.

Creating your own Vector Search Function

If you don't want to use the Python SDK, you can also write your own function to call the lookup using the requests library. Here's an example of how to do it.

import requests

def lookup(f, modality, top_k):
    f = io.StringIO(f)
    result = requests.post("https://api.vecto.ai/api/v0/space/%s/lookup" % vector_space_id,
                           data={'vector_space_id': vector_space_id, 'modality': modality, 'top_k': top_k},
                           files={'query': f},
                           headers={"Authorization":"Bearer %s" % token})
    
    return result.json()['results']

lookup("test query", "TEXT", 5)

Vector Search in the CLI

You can also perform queries using languages other than Python, with the simplest being the command line. For this example, we're going to demonstrate it using our Swagger UI interface. Navigate to Vecto API Documentation.

1. Click on the Authorize button and fill in your Vecto token. Authorize, then close the pop-up.
2. Find the Lookup: Lookup operations section. Select the one with the lookup operation, labeled as "Search for the nearest vectors to the query". This action will expand a section. Press Try it out to proceed.
3. Fill in the vector_space_id field with the id you've received. Then, set the Request body type to application/json.
4. In the request space, change "topK": 0 to "top_k": 5, remove ids, and replace the query with your own. Here's an example:

   {
   "top_k": 10,
   "query": "how to cook chicken"
   }
5. Press Execute!

You'll receive the cURL syntax, which you can copy and paste to your own terminal. If you've done it correctly, you should also see the server response for that particular query.

And that's it! In this example, you've learned how to perform a vector search and understand its capabilities. For this example, you've used a vector space that has ingested Wikipedia, but you can also ingest your own dataset for personal or company-specific searches.

If that interests you, feel free to reach out to us at https://www.vecto.ai/contactus.